A Hybrid MPI/OpenMP 3D FFT for Plane Wave First-principles Materials Science Codes

First principles electronic structure calculations based on a plane wave expansion of the wavefunctions are the most commonly used approach for electronic structure calculations in materials and nanoscience. In this approach the electronic wavefunctions are expanded in Fourier components and 3D FFTs are used to construct the charge density in real space. Efficient parallel 3D FFTs are required for many other application codes such as in fluid mechanics, climate research accelerator design, etc. Due to the large amount of communications required in 3D parallel FFTs the scaling of these application codes on large parallel machines depends critically on having a 3D FFT that scales efficiently to large processor counts. With the recent increase in the number of cores per chip/node the simple model of running one MPI process per core on large node counts results in a large number of small messages causing contention in the network and latency issues. In this paper we show that a hybrid MPI/OpenMP implementation of our 3D FFT on the Cray XT5 can significantly outperform the pure MPI version, particularly on large processor counts, by sending fewer larger messages. Our Hybrid 3D FFT has been implemented in the electronic structure code PEtot and allowed us to perform simulations of 4000 atom PbSe quantum rods on up to 21,600 cores on the Cray XT5.